Antenna Season Report Notebook¶

Josh Dillon, Last Revised January 2022

This notebook examines an individual antenna's performance over a whole season. This notebook parses information from each nightly rtp_summarynotebook (as saved to .csvs) and builds a table describing antenna performance. It also reproduces per-antenna plots from each auto_metrics notebook pertinent to the specific antenna.

In [1]:
import os
from IPython.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
In [2]:
# If you want to run this notebook locally, copy the output of the next cell into the next line of this cell.
# antenna = "004"
# csv_folder = '/lustre/aoc/projects/hera/H5C/H5C_Notebooks/_rtp_summary_'
# auto_metrics_folder = '/lustre/aoc/projects/hera/H5C/H5C_Notebooks/auto_metrics_inspect'
# os.environ["ANTENNA"] = antenna
# os.environ["CSV_FOLDER"] = csv_folder
# os.environ["AUTO_METRICS_FOLDER"] = auto_metrics_folder
In [3]:
# Use environment variables to figure out path to the csvs and auto_metrics
antenna = str(int(os.environ["ANTENNA"]))
csv_folder = os.environ["CSV_FOLDER"]
auto_metrics_folder = os.environ["AUTO_METRICS_FOLDER"]
print(f'antenna = "{antenna}"')
print(f'csv_folder = "{csv_folder}"')
print(f'auto_metrics_folder = "{auto_metrics_folder}"')
antenna = "60"
csv_folder = "/home/obs/src/H6C_Notebooks/_rtp_summary_"
auto_metrics_folder = "/home/obs/src/H6C_Notebooks/auto_metrics_inspect"
In [4]:
display(HTML(f'<h1 style=font-size:50px><u>Antenna {antenna} Report</u><p></p></h1>'))

Antenna 60 Report

In [5]:
import numpy as np
import pandas as pd
pd.set_option('display.max_rows', 1000)
import glob
import re
from hera_notebook_templates.utils import status_colors, Antenna
In [6]:
# load csvs and auto_metrics htmls in reverse chronological order
csvs = sorted(glob.glob(os.path.join(csv_folder, 'rtp_summary_table*.csv')))[::-1]
print(f'Found {len(csvs)} csvs in {csv_folder}')
auto_metric_htmls = sorted(glob.glob(auto_metrics_folder + '/auto_metrics_inspect_*.html'))[::-1]
print(f'Found {len(auto_metric_htmls)} auto_metrics notebooks in {auto_metrics_folder}')
Found 44 csvs in /home/obs/src/H6C_Notebooks/_rtp_summary_
Found 42 auto_metrics notebooks in /home/obs/src/H6C_Notebooks/auto_metrics_inspect
In [7]:
# Per-season options
mean_round_modz_cut = 4
dead_cut = 0.4
crossed_cut = 0.0

def jd_to_summary_url(jd):
    return f'https://htmlpreview.github.io/?https://github.com/HERA-Team/H6C_Notebooks/blob/main/_rtp_summary_/rtp_summary_{jd}.html'

def jd_to_auto_metrics_url(jd):
    return f'https://htmlpreview.github.io/?https://github.com/HERA-Team/H6C_Notebooks/blob/main/auto_metrics_inspect/auto_metrics_inspect_{jd}.html'

Load relevant info from summary CSVs¶

In [8]:
this_antenna = None
jds = []

# parse information about antennas and nodes
for csv in csvs:
    df = pd.read_csv(csv)
    for n in range(len(df)):
        # Add this day to the antenna
        row = df.loc[n]
        if isinstance(row['Ant'], str) and '<a href' in row['Ant']:
            antnum = int(row['Ant'].split('</a>')[0].split('>')[-1]) # it's a link, extract antnum
        else:
            antnum = int(row['Ant'])
        if antnum != int(antenna):
            continue
        
        if np.issubdtype(type(row['Node']), np.integer):
            row['Node'] = str(row['Node'])
        if type(row['Node']) == str and row['Node'].isnumeric():
            row['Node'] = 'N' + ('0' if len(row['Node']) == 1 else '') + row['Node']
            
        if this_antenna is None:
            this_antenna = Antenna(row['Ant'], row['Node'])
        jd = [int(s) for s in re.split('_|\.', csv) if s.isdigit()][-1]
        jds.append(jd)
        this_antenna.add_day(jd, row)
        break
In [9]:
# build dataframe
to_show = {'JDs': [f'<a href="{jd_to_summary_url(jd)}" target="_blank">{jd}</a>' for jd in jds]}
to_show['A Priori Status'] = [this_antenna.statuses[jd] for jd in jds]

df = pd.DataFrame(to_show)

# create bar chart columns for flagging percentages:
bar_cols = {}
bar_cols['Auto Metrics Flags'] = [this_antenna.auto_flags[jd] for jd in jds]
bar_cols[f'Dead Fraction in Ant Metrics (Jee)'] = [this_antenna.dead_flags_Jee[jd] for jd in jds]
bar_cols[f'Dead Fraction in Ant Metrics (Jnn)'] = [this_antenna.dead_flags_Jnn[jd] for jd in jds]
bar_cols['Crossed Fraction in Ant Metrics'] = [this_antenna.crossed_flags[jd] for jd in jds]
bar_cols['Flag Fraction Before Redcal'] = [this_antenna.flags_before_redcal[jd] for jd in jds]
bar_cols['Flagged By Redcal chi^2 Fraction'] = [this_antenna.redcal_flags[jd] for jd in jds]
for col in bar_cols:
    df[col] = bar_cols[col]

z_score_cols = {}
z_score_cols['ee Shape Modified Z-Score'] = [this_antenna.ee_shape_zs[jd] for jd in jds]
z_score_cols['nn Shape Modified Z-Score'] = [this_antenna.nn_shape_zs[jd] for jd in jds]
z_score_cols['ee Power Modified Z-Score'] = [this_antenna.ee_power_zs[jd] for jd in jds]
z_score_cols['nn Power Modified Z-Score'] = [this_antenna.nn_power_zs[jd] for jd in jds]
z_score_cols['ee Temporal Variability Modified Z-Score'] = [this_antenna.ee_temp_var_zs[jd] for jd in jds]
z_score_cols['nn Temporal Variability Modified Z-Score'] = [this_antenna.nn_temp_var_zs[jd] for jd in jds]
z_score_cols['ee Temporal Discontinuties Modified Z-Score'] = [this_antenna.ee_temp_discon_zs[jd] for jd in jds]
z_score_cols['nn Temporal Discontinuties Modified Z-Score'] = [this_antenna.nn_temp_discon_zs[jd] for jd in jds]
for col in z_score_cols:
    df[col] = z_score_cols[col]

ant_metrics_cols = {}
ant_metrics_cols['Average Dead Ant Metric (Jee)'] = [this_antenna.Jee_dead_metrics[jd] for jd in jds]
ant_metrics_cols['Average Dead Ant Metric (Jnn)'] = [this_antenna.Jnn_dead_metrics[jd] for jd in jds]
ant_metrics_cols['Average Crossed Ant Metric'] = [this_antenna.crossed_metrics[jd] for jd in jds]
for col in ant_metrics_cols:
    df[col] = ant_metrics_cols[col]

redcal_cols = {}
redcal_cols['Median chi^2 Per Antenna (Jee)'] = [this_antenna.Jee_chisqs[jd] for jd in jds]
redcal_cols['Median chi^2 Per Antenna (Jnn)'] = [this_antenna.Jnn_chisqs[jd] for jd in jds]   
for col in redcal_cols:
    df[col] = redcal_cols[col]

# style dataframe
table = df.style.hide_index()\
          .applymap(lambda val: f'background-color: {status_colors[val]}' if val in status_colors else '', subset=['A Priori Status']) \
          .background_gradient(cmap='viridis', vmax=mean_round_modz_cut * 3, vmin=0, axis=None, subset=list(z_score_cols.keys())) \
          .background_gradient(cmap='bwr_r', vmin=dead_cut-.25, vmax=dead_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .background_gradient(cmap='bwr_r', vmin=crossed_cut-.25, vmax=crossed_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .background_gradient(cmap='plasma', vmax=4, vmin=1, axis=None, subset=list(redcal_cols.keys())) \
          .applymap(lambda val: 'font-weight: bold' if val < dead_cut else '', subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val < crossed_cut else '', subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val > mean_round_modz_cut else '', subset=list(z_score_cols.keys())) \
          .applymap(lambda val: 'color: red' if val > mean_round_modz_cut else '', subset=list(z_score_cols.keys())) \
          .bar(subset=list(bar_cols.keys()), vmin=0, vmax=1) \
          .format({col: '{:,.4f}'.format for col in z_score_cols}) \
          .format({col: '{:,.4f}'.format for col in ant_metrics_cols}) \
          .format('{:,.2%}', na_rep='-', subset=list(bar_cols.keys())) \
          .set_table_styles([dict(selector="th",props=[('max-width', f'70pt')])]) 

Table 1: Per-Night RTP Summary Info For This Atenna¶

This table reproduces each night's row for this antenna from the RTP Summary notebooks. For more info on the columns, see those notebooks, linked in the JD column.

In [10]:
display(HTML(f'<h2>Antenna {antenna}, Node {this_antenna.node}:</h2>'))
HTML(table.render(render_links=True, escape=False))

Antenna 60, Node N05:

Out[10]:
JDs A Priori Status Auto Metrics Flags Dead Fraction in Ant Metrics (Jee) Dead Fraction in Ant Metrics (Jnn) Crossed Fraction in Ant Metrics Flag Fraction Before Redcal Flagged By Redcal chi^2 Fraction ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score Average Dead Ant Metric (Jee) Average Dead Ant Metric (Jnn) Average Crossed Ant Metric Median chi^2 Per Antenna (Jee) Median chi^2 Per Antenna (Jnn)
2459859 digital_maintenance 100.00% 100.00% 100.00% 0.00% - - 9.739351 10.937426 8.873575 9.761530 2.256655 2.836266 1.024709 1.520458 0.0272 0.0268 0.0016 nan nan
2459858 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 10.550244 11.608508 9.178806 10.007035 2.286661 2.958161 1.934819 2.878506 0.0266 0.0267 0.0007 1.129492 1.124510
2459857 digital_maintenance 100.00% 100.00% 100.00% 0.00% - - 6.107262 6.713872 1.496493 1.614892 1.205828 2.815743 5.880468 10.444087 0.0261 0.0238 0.0014 nan nan
2459856 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 15.735192 17.599416 23.176217 24.631786 6.654030 12.240390 2.841963 4.379236 0.0280 0.0284 0.0010 1.185756 1.178537
2459855 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 16.487810 17.162335 24.203167 25.488042 2.746189 4.338307 1.392116 1.913045 0.0273 0.0271 0.0008 1.187427 1.182376
2459854 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 16.537272 15.473482 18.546580 19.461940 3.869312 4.317459 4.308719 5.437847 0.0270 0.0270 0.0008 1.138912 1.136617
2459853 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 294.578223 294.003975 inf inf 4411.442915 4229.465736 10804.572349 9469.467256 nan nan nan 0.000000 0.000000
2459852 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 13.526697 16.659967 25.954312 28.118259 15.613264 19.951122 16.783592 16.591370 0.0271 0.0268 0.0009 1.161849 1.154627
2459851 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 10.815498 20.760685 26.675855 30.151158 21.516623 43.596072 13.298725 21.999729 0.0276 0.0312 0.0032 1.062239 1.053452
2459850 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 13.136291 18.230219 23.061879 25.096854 10.327660 19.918281 6.597873 17.947985 0.0265 0.0252 0.0012 1.147297 1.139243
2459849 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 15.260327 16.613896 46.561346 49.620940 7.168593 13.003060 4.855984 8.362363 0.0267 0.0254 0.0012 1.147237 1.142030
2459848 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 13.679524 15.035785 29.561586 32.088618 14.304249 21.843181 1.724090 3.354350 0.0266 0.0257 0.0011 1.154293 1.149149
2459847 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 15.771136 17.469636 27.583753 30.279560 21.865006 28.305564 0.424208 0.883354 0.0264 0.0256 0.0010 1.169570 1.167373
2459846 digital_maintenance 100.00% 100.00% 100.00% 0.00% 100.00% 0.00% 25.348015 26.168524 33.552259 35.272309 21.656050 21.028721 4.061417 3.992793 0.0267 0.0265 0.0011 1.120180 1.119672

Load antenna metric spectra and waterfalls from auto_metrics notebooks.¶

In [11]:
htmls_to_display = []
for am_html in auto_metric_htmls:
    html_to_display = ''
    # read html into a list of lines
    with open(am_html) as f:
        lines = f.readlines()
    
    # find section with this antenna's metric plots and add to html_to_display
    jd = [int(s) for s in re.split('_|\.', am_html) if s.isdigit()][-1]
    try:
        section_start_line = lines.index(f'<h2>Antenna {antenna}: {jd}</h2>\n')
    except ValueError:
        continue
    html_to_display += lines[section_start_line].replace(str(jd), f'<a href="{jd_to_auto_metrics_url(jd)}" target="_blank">{jd}</a>')
    for line in lines[section_start_line + 1:]:
        html_to_display += line
        if '<hr' in line:
            htmls_to_display.append(html_to_display)
            break

Figure 1: Antenna autocorrelation metric spectra and waterfalls.¶

These figures are reproduced from auto_metrics notebooks. For more info on the specific plots and metrics, see those notebooks (linked at the JD). The most recent 100 days (at most) are shown.

In [12]:
for i, html_to_display in enumerate(htmls_to_display):
    if i == 100:
        break
    display(HTML(html_to_display))

Antenna 60: 2459859

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Shape 10.937426 9.739351 10.937426 8.873575 9.761530 2.256655 2.836266 1.024709 1.520458

Antenna 60: 2459858

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Shape 11.608508 11.608508 10.550244 10.007035 9.178806 2.958161 2.286661 2.878506 1.934819

Antenna 60: 2459857

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Temporal Discontinuties 10.444087 6.713872 6.107262 1.614892 1.496493 2.815743 1.205828 10.444087 5.880468

Antenna 60: 2459856

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power 24.631786 15.735192 17.599416 23.176217 24.631786 6.654030 12.240390 2.841963 4.379236

Antenna 60: 2459855

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power 25.488042 17.162335 16.487810 25.488042 24.203167 4.338307 2.746189 1.913045 1.392116

Antenna 60: 2459854

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power 19.461940 15.473482 16.537272 19.461940 18.546580 4.317459 3.869312 5.437847 4.308719

Antenna 60: 2459853

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power inf 294.003975 294.578223 inf inf 4229.465736 4411.442915 9469.467256 10804.572349

Antenna 60: 2459852

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power 28.118259 13.526697 16.659967 25.954312 28.118259 15.613264 19.951122 16.783592 16.591370

Antenna 60: 2459851

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Temporal Variability 43.596072 10.815498 20.760685 26.675855 30.151158 21.516623 43.596072 13.298725 21.999729

Antenna 60: 2459850

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power 25.096854 13.136291 18.230219 23.061879 25.096854 10.327660 19.918281 6.597873 17.947985

Antenna 60: 2459849

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power 49.620940 15.260327 16.613896 46.561346 49.620940 7.168593 13.003060 4.855984 8.362363

Antenna 60: 2459848

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power 32.088618 15.035785 13.679524 32.088618 29.561586 21.843181 14.304249 3.354350 1.724090

Antenna 60: 2459847

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power 30.279560 17.469636 15.771136 30.279560 27.583753 28.305564 21.865006 0.883354 0.424208

Antenna 60: 2459846

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
60 N05 digital_maintenance nn Power 35.272309 25.348015 26.168524 33.552259 35.272309 21.656050 21.028721 4.061417 3.992793

In [ ]: